coop: stop a pair dying with one of its agents - #83

A pair is graded only when both sides return, so one agent's death loses the pair. Last night four of five pairs were lost that way. Three things had to line up wrong, and all three did.

1. Sandbox lifetimes were shorter than the agents

3600s, while a heavy reasoner generates ~4x the tokens per step and needs correspondingly more wall-clock. Agents were still working when Modal reclaimed them.

Four sites pinned it, and the explicit one wins, which is the part that bit:

Raising only the dataclass default did nothing, because adapter.py passes timeout=3600 explicitly into env_kwargs. Verified end-to-end through the live call chain (coop.py → create_git_server → ModalGitServer.create → Sandbox.create), not by reading defaults.

The git daemon matters more than it looks: it hosts the bare repo both agents push and fetch through, so one expiry breaks every git operation in the pair simultaneously. Observed as fatal: invalid object name '0ef55fd' x12 and unknown revision '18e1d15' x6 — agents chasing commits the remote no longer had.

2. Departure was announced on only one exit path

mark_exited() sat after a clean submit, so a crash or step-limit exit left the peer's has_exited() False forever. Now called from a finally, covering every in-process exit.

3. …which still cannot cover a hard kill

When the sandbox is reclaimed the process is killed outright and no finally runs. Self-reported death cannot work for the case that actually happens.

Liveness is now proven rather than death announced. Agents refresh an :alive key each step; has_exited() treats a lapsed heartbeat as gone. Silence is the signal, so detection needs no cooperation from the dead.

• ALIVE_TTL = 600 comfortably exceeds a slow step (long prompt + test run), so a working agent is never declared dead.
• An agent that has not started yet is never mistaken for one that died (_seen_alive gate).
• is_unreachable() separates killed from finished, and the message injected into the survivor's history says which occurred — a killed peer is no longer described as having "completed their work". Telling an agent something untrue about the remote is the failure this whole area exists to avoid.

Why it matters

The survivor is not misbehaving — it is running the disciplined protocol we want, gating submission on peer confirmation. With no exit signal that becomes a deadlock: one agent issued 42 sleep commands totalling 91 minutes, received zero exit notices, and outlived its own sandbox waiting.

Verification

• Five liveness cases exercised against live Redis: both alive; peer never started; graceful exit; killed peer with lapsed heartbeat; heartbeat revival.
• 447 passed, 63 skipped; ruff clean.
• Not yet exercised on a live pair — worth one flash_10 pair before relying on it.